More Algorithms for Provable Dictionary Learning

In dictionary learning, also known as sparse coding, the algorithm is given samples of the form y = Ax where x ∈ Rm is an unknown random sparse vector and A is an unknown dictionary matrix in R (usually m > n, which is the overcomplete case). The goal is to learn A and x. This problem has been studied in neuroscience, machine learning, visions, and image processing. In practice it is solved by heuristic algorithms and provable algorithms seemed hard to find. Recently, provable algorithms were found that work if the unknown feature vector x is √ n-sparse or even sparser. Spielman et al. [SWW12] did this for dictionaries where m = n; Arora et al. [AGM13] gave an algorithm for overcomplete (m > n) and incoherent matrices A; and Agarwal et al. [AAN13] handled a similar case but with weaker guarantees. This raised the problem of designing provable algorithms that allow sparsity ≫ √n in the hidden vector x. The current paper designs algorithms that allow sparsity up to n/poly(logn). It works for a class of matrices where features are individually recoverable, a new notion identified in this paper that may motivate further work. The algorithm runs in quasipolynomial time because they use limited enumeration. Princeton University, Computer Science Department and Center for Computational Intractability. Email: [email protected]. This work is supported by the NSF grants CCF-0832797, CCF-1117309, CCF-1302518, DMS-1317308, and Simons Investigator Grant. Google Research NYC. Email: [email protected]. Part of this work was done while the author was a Postdoc at EPFL, Switzerland. Microsoft Research. Email: [email protected]. Part of this work was done while the author was a graduate student at Princeton University and was supported in part by NSF grants CCF-0832797, CCF1117309, CCF-1302518, DMS-1317308, and Simons Investigator Grant. Princeton University, Computer Science Department and Center for Computational Intractability. Email: [email protected]. This work is supported by the NSF grants CCF-0832797, CCF-1117309, CCF-1302518, DMS-1317308, and Simons Investigator Grant.